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Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )S Responsive to communication(s) filed on 1 7 May 2004 . 
2a)S This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-30 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) ^3 Claim(s) 17,19 and 21 is/are allowed. 

6) |EI Claim(s) 1-16, 18. 20, 22-30 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) Q The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
1 1 )□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12)D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)D All b)Q Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. Q Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 

1 . This office action is responsive to the following communications: 
Application, filed on 09/21/2000; IDS, paper #2, filed on 09/21/2000; Request for 
reconsideration, filed 03/04/2003; Amendment A, filed 10/06/2003. 

2. Claims 1-30 are pending in this application. Claims 1, 3, 5, 7, 9, 10, 12, 
13, 15-22, 24, 25, 27, 28 and 30 are independent claims. 

3. The present title of this application is "Method for separating background 
sprite and foreground object and method for extracting segmentation mask and 
the apparatus" (as originally filed). 

Claim Rejections - 35 USC § 102 

4. The following is a quotation of the appropriate paragraphs of 35 
U.S.C. 102 that form the basis for the rejections under this section made in this 
Office action: 

A person shall be entitled to a patent unless - 

e) the invention was described in (1) an application for patent, published under section 122(b), 
by another filed in the United States before the invention by the applicant for patent or (2) a 
patent granted on an application for patent by another filed in the United States before the 
invention by the applicant for patent, except that an international application filed under the 
treaty defined in section 351 (a) shall have the effects for purposes of this subsection of an 
application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

5. Claims 1-15 are rejected under 35 U.S.C. 102(e) as being anticipated by 
Crinon et al., U.S. Patent Number 6,249,613, "Mosaic Generation and Sprite- 
Based Coding with Automatic Foreground and Background Separation", class 
382/236. 
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As per independent claim 1 , a foreground object and background sprite 
separation and extraction method for extracting . . . comprising the steps of: 
obtaining a global motion for transforming a coordinate system between a 
reference . . . ; mapping an original image corresponding to said frame into a 
reference coordinate system . . . ; generating a provisional sprite . . . ; cutting out 
a first image . . . using said global motion; obtaining a difference image between 
said first image and said original image; extracting a foreground object image . . . 
and extracting other region as a background image; mapping said background 
image . . . Crinon discloses segmenting foreground and background objects, 
coding in global motion parameters, col. 2, lines 37-55. Crinon further discloses 
reconstructing a sprite for isolating the object having the most motion in the video 
sequence, col. 3, lines 23-35. 

With respect to dependent claim 2, cutting out a second image from said 
background sprite . .. ; obtaining a difference image . . . ; extracting a foreground 
object image as a region in said difference . . . Crinon discloses in figure 8. 

As per independent claims 3 and 5, they are rejected based upon similar 
rational as above independent claim 1 respectively. 

With respect to dependent claims 4 and 6, they are rejected based upon 
similar rational as above dependent claim 2. 

As per independent claim 7, a segmentation mask extraction method . . . 
comprising the steps of: receiving a foreground mask image . . . ; providing a 
first value as an alpha value ... in each of the first macro-blocks . . . ; providing 
said first value as said alpha value to all shape pixels in each of second macro- 
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blocks . . . ; and outputting said segmentation mask. Crinon discloses first and 
second macro-blocks tagged as foreground and background objects, figure 9, 
col. 9, lines 42-67 and in col. 10, lines 1-30. 

With respect to dependent claim 8, receiving each of third macro-blocks . . 
. as said background part; and providing said first value to said third macro-block 
. . . Crinon discloses a binary segmentation map and further using a 
neighborhood of macro-blocks around a macro-block of interest, col. 9, lines 20- 
28. 

As per independent claim 9, a segmentation mask extraction . . . 
comprising the steps of: receiving a foreground mask image; generating a 
number map . . . ; initializing a foreground map; providing a predetermined value 
to each of positions in said foreground map . . . ; providing said predetermined 
value to each of position in said foreground map . . . ; and generating said 
segmentation mask . . . Crinon discloses generating macro-blocks having 
multiple local motion type vectors and further having macro-blocks as foreground 
only, background only, and foreground or background, col. 2, lines 37-67. 

As per independent claims 10 and 13, they are rejected based upon 
similar rational as above independent claim 7. 

With respect to dependent claims 1 1 and 14, they are rejected based 
upon similar rational as above dependent claim 8. 

As per independent claim 12 and 15, they are rejected based upon similar 
rational as above independent claim 9, respectively. 
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Claim Rejections - 35 USC § 103 

6. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for 
all obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described 
as set forth in section 1 02 of this title, if the differences between the subject matter sought to 
be patented and the prior art are such that the subject matter as a whole would have been 
obvious at the time the invention was made to a person having ordinary skill in the art to which 
said subject matter pertains. Patentability shall not be negatived by the manner in which the 
invention was made. 

7. Claims 16, 18, 20, 22-30 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Wang, U.S. Patent Number 6,256,409, "Method for 
Determining a Correlation between Images using Multi-Element Image 
Descriptors", class 382/170. 

As per independent claim 16, a segmentation mask extraction method . . . 
comprising the steps of: obtaining said difference image by calculating an 
absolute difference . . . initializing an energy map for each macro-block of said 
difference image; calculating energy values for said each macro-block; obtaining 
an average of said energy values; calculating a foreground ratio . . . ; and 
generating said segmentation mask . . . Wang discloses an energy map, feature 
vector or image descriptor to describe multi-band images or the correlation 
between a first image and a second image, col. 5, lines 12-59. However, it is 
noted that Wang fails to disclose representing an energy map as a macro-block 
for each different image. It would have been obvious to one of ordinary skill in 
the art at the time of the invention to include macro-blocks as image descriptors 
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in the invention of Wang to further describe the correlation of the first and second 
images for performing image matching. 

As per independent claims 18 and 20, they are rejected based upon 
similar rational as above independent claim 16. 

As per independent claim 22, a segmentation mask extraction method for 
extracting a segmentation mask . . . comprising: a first step of regarding each of 
first macro-blocks as the foreground when an energy value of said first macro- 
block which is obtained . . . ; a second step of regarding each of second macro- 
blocks as the foreground . . . Wang discloses an energy map, feature vector or 
image descriptor to describe multi-band images or the correlation of a first image 
and a second image. However, it is noted that Wang fails to disclose 
representing an energy map as a macro-block for each different image. It would 
have been obvious to one of ordinary skill in the art at the time of the invention to 
include macro-blocks as the image descriptors disclosed in Wang to further 
describe the correlation of the first and second images for performing image 
matching. 

With respect to dependent claim 23, iterating said second step for 
predetermined times. Wang further discloses re-processing and re-examining 
the image descriptors, col. 21, lines 4-17. 

As per independent claim 24, Wang further discloses comparing the 
image descriptors and categorizing the different image descriptors and belonging 
to a different image category type, col. 17, lines 32-67. However, it is noted that 
Wang fails to disclose representing an energy map as a macro-block for each 
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different image. It would have been obvious to one of ordinary skill in the art at 
the time of the invention to include macro-blocks as the image descriptors 
disclosed in Wang to describe the correlation of the first and second images for 
performing image matching. 

As per independent claims 25 and 28, they are rejected based upon 
similar rational as above independent claim 22. 

With respect to dependent claims 26 and 29, they are rejected based 
upon similar rational as above dependent claim 23. 

As per independent claims 27 and 30, they are rejected based upon 
similar rational as above independent claim 24 respectively. 

Allowable Subject Matter 

8. Claims 17, 19 and 21 are allowed. 

9. The following is a statement of reasons for the indication of allowable 
subject matter: The prior art cited in its entirety fail to render obvious dividing an 
energy value by an average to obtain the macro-blocks to represent the energy 
values in the foreground and background maps. 

Response to Arguments 

10. Applicant's arguments filed 03/04/2003 have been fully considered but 
they are not persuasive. 



Application/Control Number: 09/667,097 w Page 

Art Unit: 2672 

Applicant argues that Crinon fails to disclose the use of a provisional 
sprite. Crinon discloses the use of multiple sprites, col. 2, lines 37-55, to create a 
background sprite. Deleting foreground objects generates applicants provisional 
sprite as claimed. Crinon discloses segmenting foreground and background 
objects. Applicant argues that Crinon fails to disclose separating the foreground 
and background image using the provisional sprite and discloses only automatic 
segmentation does not require any additional frame storage and works. The text 
recites that multiple sprites are used as needed and can include the use of two 
background sprites, col. 2, lines 37-40. Furthermore it is inherent that if the sprite 
is constructed for only one object in the image and the sprite becomes more 
useable since it is related to the one object that the background image may be 
composed of multiple objects and therefore would require multiple sprites. 

Applicant argues that Crinon fails to disclose cutting out a second image 
form the background sprite by using global motion, obtaining a difference image . 
. . and extracting a foreground object image as a region in the difference image . 
. . higher than a threshold. Applicant states that in figure 8, Crinon indicates how 
the encoder distinguishes background from foreground, which Examiner 
interprets as obtaining a difference between images. Applicant further states that 
Crinon discloses global motion estimation residual are compared and if the global 
residual is greater than tagging the image as a foreground, which Examiner 
interprets as extracting a foreground object image where the difference value in 
the region is equal or higher than a threshold. 
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Applicant argues that Crinon fails to disclose first and second macro- 
blocks tagged as and foreground and background objects and extracting a 
segmentation mask and a two-stage macro-block approximation. Crinon 
indicates how the encoder distinguishes background from foreground, which 
Examiner interprets as obtaining a difference between images. Applicant states 
that Crinon refers to a process segmentation map, making regions homogeneous 
and updating the mosaic according to a new segmentation map, which Examiner 
interprets as multi process segmentation procedure. 

Applicant argues that Crinon fails to disclose providing a value to a 
position in the foreground correspond to first macro blocks . . . value to each of 
the positions in the foreground map corresponding to second macro-blocks when 
the value of the number map correspond to the second macro-block is equal to 
or larger than a second predetermined value. Applicant states that Crinon refers 
to generating macro-blocks having multiple local motion type vectors and macro- 
blocks as foreground only, which Examiner interprets as a macro-block equal to 
a first predetermined value, and foreground or background, which Examiner 
interprets as a macro-block equal to a second predetermined value. 

Applicant argues that Wang fails to disclose generating of a segmentation 
mask using foreground ratio. Wang discloses determining a correlation between 
images, col. 2, lines 10-13, the use of an energy matrix referred to as an 
orientation map, calculating the dominant orientation at each pixel, and 
combining the descriptors to form an image descriptor, col. 5, lines 12-23. 
Therefore making it obvious to use the image descriptors disclosed in Wang with 
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the mosaic images built from images of a scene and image segmentation, which 
is a correlation between images, as disclosed in Crinon. Applicant further argues 
that Wang does not describe first and second macro-block approximations. 
Crinon discloses sub-classification of the regions tagged as 1 , the foreground 
region, into 1a and 1b, for macro block coding, col. 6, lines 26-37. Crinon further 
discloses the tagged regions are coded for different image support. Wang 
discloses determining a correlation between images using an energy map. It is 
obvious that if Crinon allows for the sub-classification of the macro-blocks in the 
foreground region, the invention of Wang would allow for the coding of the sub- 
classification of the macro-blocks using energy map descriptors. 

In response to applicant's argument that the examiner's conclusion of 
obviousness is based upon improper hindsight reasoning, it must be recognized 
that any judgment on obviousness is in a sense necessarily a reconstruction 
based upon hindsight reasoning. But so long as it takes into account only 
knowledge which was within the level of ordinary skill at the time the claimed 
invention was made, and does not include knowledge gleaned only from the 
applicant's disclosure, such a reconstruction is proper. See In re McLaughlin, 
443 F.2d 1392, 170 USPQ 209 (CCPA 1971). 
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Conclusion 

1 1 . THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of 
time policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire 
THREE MONTHS from the mailing date of this action. In the event a first reply is 
filed within TWO MONTHS of the mailing date of this final action and the advisory 
action is not mailed until after the end of the THREE-MONTH shortened statutory 
period, then the shortened statutory period will expire on the date the advisory 
action is mailed, and any extension fee pursuant to 37 CFR 1 .1 36(a) will be 
calculated from the mailing date of the advisory action. In no event, however, will 
the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 



Any inquiry concerning this communication or earlier communications from 
the examiner should be directed to Motilewa A. Good-Johnson whose telephone 
number is (703) 305-3939. The examiner can normally be reached on Monday - 
Friday 8:30 AM - 5:00 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the 
examiner's supervisor, Mike Razavi can be reached on (703) 305-4713. The fax 
phone number for the organization where this application or proceeding is 
assigned is 703-872-9306. 
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Information regarding the status of an application may be obtained from 
the Patent Application Information Retrieval (PAIR) system. Status information 
for published applications may be obtained from either Private PAIR or Public 
PAIR. Status information for unpublished applications is available through 
Private PAIR only. For more information about the PAIR system, see http://pair- 
direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll- 
free). 



Motilewa A. Good-Johnson 

Examiner 
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